Learning Sequential Latent Variable Models from Multimodal Time Series Data

Sequential modelling of high-dimensional data is an important problem that appears in many domains including model-based reinforcement learning and dynamics identification for control. Latent variable models applied to sequential (i.e., latent models) have been shown be a particularly effective probabilistic approach solve this problem, especially when dealing with images. However, application areas (e.g., robotics), information from multiple sensing modalities available -- existing methods not yet extended effectively make use such multimodal data. Multimodal sensor streams can correlated useful manner often contain complementary across modalities. In work, we present self-supervised generative framework jointly learn state representation the respective dynamics. Using synthetic real-world datasets robotic planar pushing task, demonstrate our leads significant improvements prediction quality. Furthermore, compare common baseline concatenating each modality space show principled formulation performs better. Finally, despite being fully self-supervised, method nearly as supervised relies on ground truth labels.